Grid based data mobility

ABSTRACT

A data migration system and method are disclosed for migrating data from a source server to a target server. The system includes an index containing a plurality of data migration operations in a normalized data model, each data migration operation being stored in association with an attribute, and a data mover communicably connected to the index. The data mover is adapted to move data from the source server to the target server in accordance with the data migration operations contained in the index. The data mover has an attribute corresponding to the associated attribute of at least one data migration operation contained in the index, the attribute indicating the type of data migration operations that can be performed by the data mover.

The present application is a continuation of U.S. patent applicationSer. No. 13/914,728, filed Jun. 11, 2013 which in turn claims priorityto United Kingdom Patent Application Serial Number 1214119.8 entitled“GRID BASED DATA MOBILITY”, filed Aug. 7, 2012 with the United KingdomIntellectual Property Office, the contents of each applicationidentified above are incorporated by reference herein in its entirety.

BACKGROUND

Field of the Invention

The invention relates to a system and process for moving data from asource server to a target server in a more scalable and flexible manner.

Background of Invention

The migration of large volumes of data between different physicalsystems and data models is becoming progressively more important toindustry. Such migrations may be necessary to move data from legacyhardware or data models that are now obsolete. Migrations are alsonecessary to provide backup of data or to move data to a more accessiblephysical location.

Data migration has previously been performed using labor-intensivemanual data collection and transformation processes that must betailored to each individual migration. This has made the processexpensive, slow and prone to error.

Any type of data mobility operation involves a high number of source andtarget resources that must be manipulated to maximize the efficiency ofthe data movement and reduce the overall duration of the operation.Large data infrastructures are built to provide for many users accessingthe data and creating new data in a storage infrastructure.

Data migration operations tend to be a single thread, in contrast to theabove multiple thread infrastructures. This single-thread operation doesnot fully optimize the source infrastructure asset to enable a fast andefficient data movement.

SUMMARY

According to an aspect of the invention, there is provided a datamigration system for migrating data from a source server to a targetserver, including: an index containing a plurality of data migrationoperations in a normalized data model, each data migration operationbeing stored in association with an attribute; and a data movercommunicably connected to the index, the data mover being adapted tomove data from the source server to the target server in accordance withthe data migration operations contained in the index; wherein: the datamover has an attribute corresponding to the associated attribute of atleast one data migration operation contained in the index, the attributeindicating the type of data migration operations that can be performedby the data mover; the index is adapted to acquire the attribute of thedata mover and to transmit to the data mover a data migration operationhaving the attribute corresponding to the attribute of the data moverfrom among the data migration operations contained in the index; and thedata mover is adapted to move data from the source server to the targetserver in accordance with the transmitted data migration operation.

The present invention provides a logic mechanism that allows the massivescalability of data mobility and data movement operations by the use ofa central job repository and abstracted data mover applications.

To enable many threads of data migration operations, the presentinventors realized that many abstracted data movers may be used tofacilitate many requests to the source infrastructure. This structureeffectively drives up aggregate throughput. However, multiple datamovers may not always be required and the appropriate number of datamovers will vary between migrations. The system of the present inventionallows data movers to be added to and removed from the system asrequired, so that the throughput of the system can be matched to itsworkload.

Preferably, the data mover includes a data extraction command generationengine adapted to generate a data extraction command in a sourcelanguage of the source server from the transmitted data migrationoperation. Suitably, the data mover includes a data commit commandgeneration engine adapted to generate a data commit command in a targetlanguage of the target server from the transmitted data migrationoperation.

Preferably, the data mover is adapted to send a completion report to theindex after the transmitted data migration operation is completed.

Suitably, the index is adapted to determine whether all data migrationoperations having the attribute corresponding to the attribute of thedata mover have been completed in a current phase of data migration,after the transmitted data migration operation is completed; and if notall data migration operations having the attribute have been completedin the current phase of data migration then the index is adapted totransmit another data migration operation having the attribute fromamong the data migration operations contained in the index; and if alldata migration operations having the attribute have been completed inthe current phase of data migration then the index is adapted toinstruct the data mover to enter a paused state.

More preferably: the data mover has a plurality of attributes and eachdata migration operation is associated with a plurality of attributes;the attributes of the data mover correspond to the attributes of atleast one data migration operation contained in the index; and the indexis adapted to transmit to the data mover a data migration operationhaving attributes corresponding to the attributes of the data mover fromamong the data migration operations contained in the index.

Suitably, the system includes a plurality of data movers havingdifferent attributes, wherein the index distributes data migrationoperations to data movers having attributes respectively matching thoseof the data migration operations.

The inventors have discovered that the added complexity associated withusing multiple data movers, which is due to the management andconfiguration of many elements, avoiding contention and avoidingrepeated operations, is greatly outweighed by the increase in aggregatethroughput provided by the invention.

According to another aspect of the invention, there is provided a methodfor migrating data from a source server to a target server, including:storing a set of data migration operations in a normalized data model inan index, each data migration operation being stored in association withan attribute; storing an attribute in a data mover communicablyconnected to the index, the attribute indicating the type of datamigration operations that can be performed by the data mover; acquiringthe attribute stored in the data mover at the index; transmitting fromthe index to the data mover a data migration operation having anassociated attribute corresponding to the attribute stored in the datamover; converting the data migration operation into a data extractioncommand in a source language of the source server; transmitting the dataextraction command to the source server; converting the data migrationoperation into a data commit command in a target language of the targetserver; transmitting the data commit command to the target server; andmoving data from the source server to the target server.

Preferably, a plurality of data movers having different attributes areconnected to the index, the method further including: estimating thetime required to complete movement of the data from the source server tothe target server; and connecting an additional data mover to the indexor activating an additional data mover if the estimated time exceeds apredetermined upper threshold.

Suitably, a plurality of data movers having different attributes areconnected to the index, the method further including: estimating thetime required to complete movement of the data from the source server tothe target server; and disconnecting a data mover from the index ordeactivating a data mover if the estimated time falls below apredetermined lower threshold.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The following detailed description, given by way of example and notintended to limit the invention solely thereto, will best be appreciatedin conjunction with the accompanying drawings, in which:

FIG. 1 illustrates a data migration system according to an embodiment ofthe invention;

FIG. 2 is a flow diagram showing the operation of a data mover toperform a data migration operation in an embodiment of the invention;

FIG. 3 is a flow diagram showing the operation of the index to perform adata migration operation in an embodiment of the invention;

FIG. 4 is a schematic diagram showing the main components of the datamover; and

FIG. 5 illustrates an exemplary computer architecture 1100 on which thedata migration system of the invention can be implemented.

The drawings are not necessarily to scale. The drawings are merelyschematic representations, not intended to portray specific parametersof the invention. The drawings are intended to depict only typicalembodiments of the invention. In the drawings, like numbering representslike elements.

DETAILED DESCRIPTION

Detailed embodiments of the claimed structures and methods are disclosedherein; however, it can be understood that the disclosed embodiments aremerely illustrative of the claimed structures and methods that may beembodied in various forms. This invention may, however, be embodied inmany different forms and should not be construed as limited to theexemplary embodiments set forth herein. Rather, these exemplaryembodiments are provided so that this disclosure will be thorough andcomplete and will fully convey the scope of this invention to thoseskilled in the art. In the description, details of well-known featuresand techniques may be omitted to avoid unnecessarily obscuring thepresented embodiments.

FIG. 1 provides an overview of an index—data mover relationshipaccording to an embodiment of the invention. With reference to FIG. 1,the source server 300 contains source data, which is to be migrated totarget data in the target server 400.

The index 200 contains an index of the source data in the source server300. The index 200 uses a normalized data model that is independent ofthe format in which data is actually held in the source server 300. Theindex 200 is generated using automated data collectors and importers.

The data mover 100 is a component that moves data from the source server300 to the target server 400 based on the information about the sourcedata contained in the index 200. The system may contain any number ofdata movers 100. The data mover 100 takes a normalized workload from theindex 200. The term “normalized” means that all data in the index 200conforms to a single standard data model, which is understood by thedata movers 100. In other words, the index 200 is agnostic to the formatand type of data held in the source and target servers. The index 200 isunaffected by the specific database products used on the source andtarget servers. All product-specific translation and management isconducted on the data mover 100.

In this embodiment of the invention, the index 200 is a central indexwhich stores all required data movement operations. These have beenmathematically created and processed, each with a number of attributessuch as source location, target destination, platform type and datatype. These attributes are tagged to every migration operation, meaningthat there is metadata controlling each data operation. The migrationoperations are then entered as eligible operations on the central index200, effectively marking each migration operation as a piece of workthat should be completed if the data mover 100 aligns with the requiredattributes for that particular migration operation.

For example, a data migration operation may be stored in the index 200having the following attributes: Source A, Target B, Type C and PlatformD. Many data movers 100 may be provided, only a subset of which have theattributes Source A, Target B, Type C and Platform D. Any one of thatsubset of data movers may perform the data migration operation. The datamover 100 can be pre-configured with the attributes in a control file,or the configuration of the attributes of the data mover 100 can becompleted by the central index 200.

This allows a many data mover 100 to one index 200 relationship. As newdata movers communicate with the central index 200 and make themselvesavailable for work, migration operations are allocated to the data mover100. The data mover 100 on connection automatically registers itselfwith the associated attributes, if these match migration operations withthe equivalent attributes then those migration operations are madeavailable to be allocated to the data mover 100.

Once the migration operations are allocated to the data mover 100, thedata mover 100 has no further reliance on the central index 200 otherthan pushing progress and success reports to the index 200 to ensurethat the central index 200 is aware of the completion of all operations.The data mover 100 secures the source and target point relationshipswith the associated infrastructure, i.e. the connections between thedata mover 100 and the source server 300 and between the data mover 100and the target server 400. The data mover 100 also handles alltranslation of required operations to the languages of the source andtarget servers, referred to as the source and target languages.

In other words, the central index 200 stores an array of migrationoperations in a normalized data model. Each data mover 100 convertsmigration operations having attributes matching its own into theappropriate source and target languages depending on its attributes. Forexample, the data mover 100 described above would produce a request toretrieve data in the language of source A based on the entry for thedata migration operation in the index 200 and send the request to theappropriate source server A, to which the data mover 100 is connected.The data mover 100 thereby retrieves the relevant data from the sourceserver 300.

Subsequently, the data mover 100 would generate a request to commit,i.e. upload data in the language of target B, again based on the entryfor the data migration operation in the index 200. This request is sentto target server B to which the data mover 100 is connected and therelevant data is thereby committed in the appropriate format for targetserver B.

Typically, each data mover 100 has translation logic for converting adata migration operation in the normalized data model, i.e. language, ofthe index 200 to requests in the languages of one or more particularsource servers and one or more particular target servers respectively.The translation logic constitutes data extraction command generation anddata commit command generation engines. As a result, the index onlyneeds to ensure that the data migration operations are sent toappropriate data movers to ensure that the operations are translatedinto the correct languages.

Once each data mover 100 has completed the current migration operation,then it can either take additional workload or pause its operation andbe removed from the active configuration. This optional pausing of datamovers ensures that the aggregate capability can be flexed as the sourceenvironment is reduced through data migration.

As the central index 200 holds normalized, common attributes for eachmigration operation, the scale out capability of the data movers is thenumber of individual attribute profiles and the number of migrationoperations within the index 200.

The enhanced scalability of the system of the embodiment is due to thefact that the data movers are all independent automated componentshaving well defined attributes. This means that additional data moverscan be added to the system to increase throughput without interferingwith the operation of the system. The new data movers simply receivework from the index 200 in the same way as the pre-existing data movers,thereby relieving the load on the pre-existing data movers withoutinterrupting their operation. The index 200 immediately knows how toallocate work to new data movers by matching the attributes of availabledata migration operations to the attributes of the new data movers.

FIG. 1 illustrates the scalability of the data mover layer. The figureshows available migration operations with specific attributes, and thescaling of data movers in line with optimizing aggregate data transferbandwidth and reducing total duration. The solution of the embodimentensures that the data mobility layer always exceeds the aggregatebandwidth capability of the source environment, meaning that the datamobility layer is never the bottleneck to data movement operations. Datamovers can be adjusted to different identities as required. Data moverscan also be dynamically added and removed as required. This allowsscaling of the migration capability as discussed above.

A process of data migration according to an embodiment of the inventionis described below and is illustrated in FIGS. 2 and 3. FIG. 2 shows theoperations of the data mover 100, whereas FIG. 3 shows the operations ofthe index 200.

Once the migration operation setup and configuration on the centralindex has been completed, a workload list is available for migrationoperations. All eligible data is packaged into a normalized workloadlist of required operations. Once this list is complete, then the datamovers can be configured and started.

After the data movers have connected to the central index at step S10,their attributes are defined and checked against the attributes of eachdata migration operation on the workload list at steps S30 and S32. Thedata mover can be switched on, i.e. its data migration functions areactivated, and when this is completed the appropriate workload, i.e. adata migration operation, will be allocated to the data mover at stepS34. The workload is received by the data mover at step S12. Thisworkload is essentially a list of the data objects residing in thesource server, provided by the index in the normalized language of theindex. The data mover then translates this normalized content into theextraction command for the source environment and the commit command forthe target environment at steps S14 and S16. The data mover thenorchestrates the process of moving data according to the extraction andcommit commands at step S18 and reports back to the central index sothat a central log of the operation is retained.

Once the migration operation is complete, the data mover reportscompletion of the data migration operation to the index at step S20. Theindex receives the operation completion report from the data mover atstep S36 and records completion of the operation at step S38.

The data mover will progress to all other operations within the specificphase of migration if any such operations remain. Provided that not alldata migration operations in the current phase have been completed, theindex returns to step S32 and allocates more work to the data mover.Once all data migration operations in the current phase are complete theindex instructs the data mover to move into a paused state at step S40,as the data mover is waiting to acquire the next eligible migrationoperation. This ensures the central index is updated as to the status ofevery migration object stored in the source environment. Operations canbe tracked and rerun if required throughout the migration program.

FIG. 4 shows the main components of the data mover 100. Thecommunication interface 110 communicates with the source server 300, thetarget server 400 and the index 200. The data extraction commandgeneration engine 106 generates the data extraction command in thelanguage of the source server 300 based on a data migration operationreceived from the index 200 via the communication interface 110. Thedata commit command generation engine 108 generates the data commitcommand in the language of the target server 400 based on the datamigration operation received from the index 200.

The data storage 102 stores data being moved between the source server300 and the target server 400. The attribute storage 104 stores theattributes of the data mover 100 that are compared to attributes of thedata migration operations in the index 200.

Additional data movers can be started at any point to increase themigration streams and the aggregate available mobility bandwidth. Inaddition data movers can be removed from the active configuration oncethey have returned to the pause state after the completion of amigration operation. These operations can be scheduled from the centralindex 200, ensuring that the data movers can be dynamically enabled anddisabled based on policy.

In one embodiment, the data migration system estimates the time requiredto complete the movement of all data involved in the current datamigration from the source server 300 to the target server 400. This maybe achieved by obtaining the average rate of data transfer from thestart of the data migration through each data mover 100 from the datamovers, and adding these average throughputs to obtain the total averagethroughput of the data migration system. Dividing the volume of datathat remains to be migrated by the total average throughput then yieldsthe estimated time to completion.

The data migration system compares the estimated time to completion witha desired time to completion for the data migration, which is input tothe system in advance. If the estimated time to completion is higherthan the desired time then the system activates currently deactivateddata movers or connects additional data movers to the system. Thisincreases the overall rate of data transfer and ensures timelycompletion of the migration.

Conversely, if the estimated time to completion is less than the desiredtime by more than a present amount, the system deactivates one or moredata movers or disconnects one or more data movers from the index. Thisconserves power and system resources when they are not required.

In the above embodiment, the index 200 may send a request for attributesto a data mover 100 and the data mover 100 may transmit its attributesto the index 200 in response to the request. However, it is alsopossible for the data mover 100 to transmit its attributes to the index200 unprompted on connection to the index 200 or at regular intervals.The index 200 may also store a list of all data movers 100 connected tothe index and their attributes.

In the above embodiment the index 200 allocates and sends data migrationoperations to the data movers 100. However, the index 200 may first waitfor a work request to be sent from a data mover 100 before allocating adata migration operation to that data mover 100.

The above method allows the flexible capability for large scale datamobility to be scaled to the size and capability of any sourceinfrastructure. This flexible style of grid based data movement meansthat larger volumes of data having more complex data types and platformscan be migrated than was possible in the prior art. The number of datamovers used in the invention is not limited and can be selected freelyin line with the size and planned duration of the migration project.

The design of the system according to this embodiment allows a singlecentral index or repository to hold all associated metadata for themigration project. This allows a single point of management and controlfor the management of all data movers.

In this embodiment, the data mover interacts with a client module forthe source and target software. This allows the application of theprocess to other applications requiring scalable data mobility. Possibleapplications of the invention include providing indexed backup/archivingof data content, the migration of large scale physical-virtual serverenvironments, and the mobility of storage assets to more moderntechnology.

FIG. 5 illustrates an exemplary computer architecture 1100 by which thedata migration system and in particular an index or a data moveraccording to the invention may be implemented. Computer architecture1100 may be or form part of a desktop computer or a laptop computer, aserver or any similar computer device, but the index is preferablyimplemented as a stand alone server.

The computer architecture 1100 may interface to external devices such asthe source server 300 or the target server 400 through a modem ornetwork interface 1102, such as an analogue modem, ISDN modem, cablemodem, token ring interface, or satellite transmission interface. Asshown in FIG. 5, the computer architecture 1100 includes a processingunit 1104, which may be a conventional microprocessor, such as an IntelPentium microprocessor, an Intel Core Duo microprocessor, or a MotorolaPower PC microprocessor, which are known to one of ordinary skill in thecomputer art. System memory 1106 is coupled to the processing unit 1104by a system bus 1108. System memory 1106 may be a DRAM, RAM, static RAM(SRAM) or any combination thereof. Bus 1108 couples processing unit 1104to system memory 1106, to non-volatile storage 1110, to graphicssubsystem 1112 and to input/output (I/O) controller 1114. Graphicssubsystem 1112 controls a display device 1116, such as a liquid crystaldisplay, which may be part of the graphics subsystem 1112. The I/Odevices 1118 may include one or more of a keyboard, disk drives,printers, a mouse, a touch screen and the like as known to one ofordinary skill in the computer art.

The index or data mover control software will normally be stored on thenon-volatile storage 1110. Thus, it may be stored on the machine's harddrive, or possibly on an externally connectable storage medium, such asa USB memory stick or a CD. These two devices would then constitute partof the I/O devices shown as item 1118 in FIG. 5. The non-volatilestorage may also store indexing data forming part of the index.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiment, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A computer system for migrating data from asource server to a target server, the computer system comprising: one ormore computer processors, one or more non-transitory computer-readablestorage media, and program instructions stored on the one or morecomputer-readable storage media for execution by at least one of the oneor more computer processors, the program instructions comprising:program instructions to store a set of data migration operations in anormalized data model in a single index, each data migration operationbeing stored in association with an attribute; program instructions tostore an attribute profile in a data mover communicably connected to thesingle index, the attribute profile indicating the type of datamigration operations that are performed by the data mover; programinstructions to acquire, by the single index, the attribute profilestored in the data mover; program instructions to transmit, from thesingle index to the data mover, a data migration operation from the setof data migration operations stored in the single index, wherein theattribute profile of the data mover matches an attribute of thetransmitted data migration operation; program instructions to convertthe transmitted data migration operation into a data extraction commandin a source language of the source server; program instructions totransmit the data extraction command to the source server; programinstructions to convert the transmitted data migration operation into adata commit command in a target language of the target server; programinstructions to transmit the data commit command to the target server;and program instructions to move data from the source server to thetarget server.
 2. The computer system according to claim 1, wherein aplurality of data movers having different attributes are connected tothe single index, the method further comprising: program instructions toestimate the time required to complete movement of the data from thesource server to the target server; and program instructions to connectan additional data mover to the single index or activating an additionaldata mover if the estimated time exceeds a predetermined upperthreshold.
 3. The computer system according to claim 1, wherein aplurality of data movers having different attributes are connected tothe single index, the method further comprising: program instructions toestimate the time required to complete movement of the data from thesource server to the target server; and program instructions todisconnect a data mover from the single index or deactivating a datamover if the estimated time falls below a predetermined lower threshold.4. The computer system according to claim 1, wherein the single indexfurther comprises an index of the data on the source server.
 5. Thecomputer system according to claim 1, wherein the attribute of the datamigration operation comprises a source location, a target destination, aplatform type, a data type, or some combination thereof.
 6. The computersystem according to claim 1, wherein the attribute profile of the datamover comprises a source location, a target destination, a platformtype, a data type, or some combination thereof.
 7. A computer system formigrating data from a source server to a target server, the computersystem comprising: one or more computer processors, one or morenon-transitory computer-readable storage media, and program instructionsstored on the one or more computer-readable storage media for executionby at least one of the one or more computer processors, the programinstructions comprising: program instructions to store a set of datamigration operations in a normalized data model in a central index, eachdata migration operation being stored in association with one or moreattributes; program instructions to acquire, by the central index, anattribute profile stored in one of a plurality of data moverscommunicably connected to the central index; program instructions totransmit a data migration operation from the set of data migrationoperations stored in the central index, the data migration operation istransmitted from the central index to one of the plurality of datamovers having an attribute profile matching the one or more attributesof the data migration operation; program instructions to convert thetransmitted data migration operation into a data extraction command in asource language of the source server; program instructions to transmitthe data extraction command to the source server; program instructionsto convert the transmitted data migration operation into a data commitcommand in a target language of the target server; program instructionsto transmit the data commit command to the target server; and programinstructions to move data from the source server to the target server.8. The computer system according to claim 7, further comprising: programinstructions to estimate the time required to complete movement of thedata from the source server to the target server; and programinstructions to connect an additional data mover to the central indexbased on a comparison between the estimated time required to completemovement of the data and a predetermined threshold.
 9. The computersystem according to claim 7, further comprising: program instructions toestimate the time required to complete movement of the data from thesource server to the target server; and program instructions to activatean additional data mover to the central index based on a comparisonbetween the estimated time required to complete movement of the data anda predetermined threshold.
 10. The computer system according to claim 7,further comprising: program instructions to estimate the time requiredto complete movement of the data from the source server to the targetserver; and program instructions to disconnect a data mover from thecentral index based on a comparison between the estimated time requiredto complete movement of the data and a predetermined threshold.
 11. Thecomputer system according to claim 7, further comprising: programinstructions to estimate the time required to complete movement of thedata from the source server to the target server; and programinstructions to deactivate a data mover from the central index based ona comparison between the estimated time required to complete movement ofthe data and a predetermined threshold.
 12. The computer systemaccording to claim 7, wherein the central index further comprises anindex of the data on the source server.
 13. The computer systemaccording to claim 7, wherein the one or more attributes of the datamigration operation and the attribute profile of the data movercomprises a source location, a target destination, a platform type, adata type, or some combination thereof.